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ABSTRACT 



The Academy for External Facilitators is a significant 
component of AEL's Comprehensive School Reform Demonstration and is the 
subject of this report. External facilitators are usually assigned by state 
departments of education to assist low-performing schools. This report 
summarizes evaluation of the second summer institute of the Academy for 
External Facilitators, convened August 2-5, 2000. Forty-seven participants 
from Kentucky, Tennessee, Virginia, and West Virginia attended the institute. 
Participants tended to be pleased with the institute and perceived presenters 
to be competent and their presentations professional and appealing. External 
facilitators participating in both the 1999 and 2000 institutes showed more 
skill and knowledge in 2000 with regard to school reform models and 
strategies, technological abilities, team building, group process, resource 
identification, the change process, situational analysis, and collaborative 
leadership. Recommendations included providing participants with more time to 
interact with presenters, focusing future institutes on areas in which 
external facilitators report the least expertise, and continuing to offer 
participants access to Academy resources and events to support ongoing work. 
There are 14 tables and one figure. Appendices include a Stages of Concern 
Questionnaire, an External Facilitator Self-Assessment Questionnaire, session 
evaluation forms, an Overall Institute Evaluation Form, the 2000 Institute 
Agenda, and the Completed Evaluation Standards Checklist. (RT) 
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EXECUTIVE SUMMARY 



As part of the FY 98 Labor-HHS-Education Appropriations Act (PL 105-78), Congress 
designated $150 million to promote comprehensive school reform. Much of these Comprehensive 
School Reform Demonstration (CSRD) funds flowed through states via a competitive grant process 
to schools and districts, particularly those receiving Title I assistance, interested in implementing 
schoolwide comprehensive reform. In addition, $4 million was allocated to Regional Educational 
Laboratories to assist states, districts, and schools in the implementation of the CSRD program. 

One significant component of AEL’s CSRD program, and the subject of this report, is an 
Academy for External Facilitators. External facilitators are individuals usually assigned by state 
departments of education to assist low-performing schools — in this case, those implementing CSRD- 
funded reform initiatives. The Academy is intended to support the efforts of external facilitators as 
they begin their work with CSRD-funded schools. 

This report summarizes evaluation of the second summer institute of the Academy for 
External Facilitators, convened August 2-5, 2000, at the Arlie Center in Warrenton, Virginia. Forty- 
seven participants attended the 2000 institute. Attendance by state is as follows: Kentucky, 5; 
Tennessee, 14; Virginia, 19; and West Virginia, 9. Participants were individuals selected to serve 
as external facilitators in schools implementing CSRD reform and various state departments of 
education staff. In addition, 10 AEL staff were available to facilitate the event. 

Participants tended to be pleased with the institute, which they reported provided them with 
useful information and materials. Presenters were perceived to be competent and their presentations 
professional and appealing. External facilitators participating in both the 1999 and 2000 institutes 
were significantly and substantially more skilled and knowledgeable in 2000 with regard to school 
reform, reform models and strategies, technological abilities, CSRDnet, team building, group 
process, resource identification, network development, the change process, environmental scanning, 
and situational analysis. And to a lesser extent, external facilitators were also more skilled and 
knowledgeable about collaborative leadership. 

Overall, attendees at the 2000 institute were less concerned about their roles as external 
facilitators than they had been the previous year. Although they remained most concerned about their 
awareness of the innovation and collaboration with others, they also continued to have a high 
concern for information about external facilitation. External facilitators were least concerned about 
the consequences or outcomes of their work, as they had been the previous year. Individuals were 
most concerned with either awareness and information about the innovation, or awareness and 
collaboration with others. 

External facilitators who had participated in the 1999 and 2000 institutes were significantly 
and substantially less concerned about their awareness of the innovation, information levels, personal 
issues, and management considerations than they had been in 1999. Their levels of concern about 
collaboration and refocusing remained fairly stable in 1999 and 2000. 

Recommendations included providing participants with more time to interact with presenters, 
focusing future institutes on areas in which external facilitators report the least expertise, and 
continuing to offer participants access to Academy resources and events to support their ongoing 
work. 

iii 
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INTRODUCTION 



Federal CSRD Legislation 

School improvement is increasingly viewed as an ongoing and comprehensive process. 
Recent legislation has encouraged the adoption of such a view. In 1994, Congress revised the 
regulations to allow schools where 50% or more students qualify for free or reduced lunch to use 
their Title I funds for whole school improvement rather than for programs targeted solely at specific 
groups of students (American Institutes for Research, 1 999). Later, as part of the FY 98 Labor-HHS- 
Education Appropriations Act (PL 105-78), and again in 1999, Congress designated $ 1 50 million 
to promote comprehensive school reform. Much of these Comprehensive School Reform 
Demonstration (CSRD) funds flowed through states via a competitive grant process to schools and 
districts interested in implementing schoolwide comprehensive reform. Of these monies, $120 
million came from Elementary and Secondary Education Act (ESEA) Title I funds allocated by 
formula to states to allow current Title I schools to adopt comprehensive school reform programs. 
The Fund for the Improvement of Education (FIE) allocated $25 million to flow to states by school- 
age population to allow non-Title I schools to implement comprehensive school reform programs. 
In addition, $4 million was allocated to Regional Educational Laboratories to assist states, districts, 
and schools in the implementation of the CSRD program. 

According to the New American Schools (NAS) organization ( 1 999), which has developed 
several of the school wide reform models adopted by CSRD-funded schools, “the success or failure 
of the CSRD Program has enormous implications for public education in the country’s poorest 
neighborhoods.” Title I of the ESEA of 1965, which provides federal funding for disadvantaged 
students, is due for congressional reauthorization in the year 2000. “To encourage research-based 
and replicable best practices,” Congress may reauthorize Title I based upon lessons learned from 
CSRD (NAS, 1999). 

AEL CSRD Program 

AEL, Inc. began assisting the four states in its region (Kentucky, Tennessee, Virginia, and 
West Virginia) in early 1998 as part of the effort to support state departments of education in their 
implementation of CSRD. Assistance to these state departments of education has included 
consultation and technical assistance concerning the development of procedures and criteria to select 
competitively the schools that would receive assistance under this program. Also, AEL has 
collaborated with state departments of education in developing procedures for guiding interested 
districts and schools though the process of assessing their needs, selecting among research-based 
reform models and strategies, or choosing to develop their own research-based comprehensive 
program. Also in collaboration with state departments, AEL has developed and provided workshops 
and technical assistance to schools and districts preparing their applications for grants under this 
program. Such workshops have included information about candidate reform models, assistance 
with compiling aggregate components to create a comprehensive program, and guidance on 
evaluating the effectiveness of programs implemented. 

Another facet of AEL’s CSRD program has been to assist states and schools in the evaluation 
of the effectiveness of programs implemented with CSRD funds. Participating CSRD schools are 
required to evaluate their progress toward the nine CSRD implementation criteria (evaluation itself 
being one of the nine criteria). AEL CSRD services have additionally included two national research 
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symposia on the CSRD program and schoolwide reform, and several independent and collaborative 
research efforts. 

Yet another AEL CSRD goal is to develop electronic networks of schools, districts, and 
external facilitators involved in CSRD. The purpose of such networks is to provide participants with 
opportunities to share experiences and information, connect with resources, and access relevant 
information. Moreover, the electronic networks provide AEL a venue for offering ongoing support 
to CSRD school sites and external facilitators. These networks are also intended to provide AEL 
with information regarding contributions to successful implementation of CSRD programs for the 
purpose of informing the wider education community. 

AEL has established two types of electronic networks in which staff at CSRD sites, external 
facilitators, and site researchers assisting with formative evaluation of CSRD schools may 
participate. CSRDnet is a web-based electronic communication forum providing moderated 
discussion and conferencing capabilities for external facilitators and CSRD sites. Participants are 
assigned to several discussion groups based on the comprehensive school reform (CSR) model they 
are implementing and the state in which they work. A form is provided for participants to enter 
information about themselves and their undertakings; a similar form is provided for participants to 
complete profiles of their schools. In addition, a moderator posts articles and links relevant to CSR. 

The second electronic medium consists of listservs in three of AEL’s four states (Kentucky, 
Virginia, and West Virginia). As with CSRDnet, a moderator facilitates discussion and information 
sharing, and participants include external facilitators and staff members at CSRD sites. 

Academy for External Facilitators 

Another significant component of AEL’s CSRD program, and the subject of this report, is 
an Academy for External Facilitators. External facilitators are individuals usually assigned by state 
departments of education to assist low-performing schools — in this case, those implementing CSRD- 
funded reform initiatives. The use of external facilitators in such a role may have implications for 
the structure of future assistance to low-performing schools. 

AEL’s Academy offers training and support as external facilitators execute five challenging 
roles in school reform: using the knowledge base on effective practices leading to improved student 
performance; modeling collaboration and joint problem-solving; collecting, organizing, and 
analyzing school data to make decisions; brokering and recommending resources; and acting as 
agents of change in contexts in which facilitators possess no authority to direct change. 

The Academy is based on an action research leam-as-you-practice model. AEL convened 
a team of national experts in school change to design the Academy and its sessions. After an initial 
5-day residential institute in 1 999, Academy participants were electronically networked to gain 
information and support as they worked at school sites. Participants honed their skills further at 
subsequent meetings to share their experiences and receive further training. The phases of the 
Academy are briefly described below. 

Phase I: Phase I consisted of a 1-day introductory state department meeting for each state 
in AEL’s region during which participants gathered to discuss state-specific issues and Academy 
orientation. The second part of Phase I was a 6-day residential institute in August 1 999 during which 
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participants attended sessions conducted by national experts on such topics as whole-school reform, 
the change process, formative evaluation, making informed decisions based on data, and 
organizational leadership and development. 

Phase II: Phase II provided support to external facilitators participating in the Academy as 
they worked at their sites. Phase II linked participants through technology — moderated listservs, 
conferencing areas, and chat rooms with facilitated conversations. External facilitators at sites in 
each state were connected to receive both general and state-specific information on resources, issues 
of concern, and strategies for change. In addition, sites were connected via electronic forums across 
states according to selected models. 

Phase III: Phase III consisted of follow-up state meetings for participants, who attended at 
least one 1 -day meeting for reflection and assessment of progress. 

Phase IV: Phase IV consisted of a second residential institute. Participants from the first 
cohort attended a meeting that coincided with the first residential institute of the next cohort of 
external facilitators. Second-year attendees received follow-up training and shared expertise among 
themselves and new participants. This report summarizes evaluation data from the Phase IV 
residential institute convened August 2-5, 2000, at the Arlie Center in Warrenton, Virginia. 

Goals and Objectives of the 2000 Academy Institute 

The goals of the second Academy institute were to help participants 

• broaden their understanding of school reform and the change process 

• increase their repertoire of strategies for helping schools build capacity for 
continuous improvement 

• inform their state’s future reform efforts through the distillation of lessons learned 

• expand their resource networks 

In addition, AEL staff outlined daily objectives for the institute. These were to 

• clarify the purpose and scope of the 2000 summer institute (Wednesday, August 2) 

• capture lessons learned from the first year of CSRD implementation (Wednesday, 
August 2) 

• expand understanding of how change is planned, managed, and implemented 
(Thursday, August 3) 

• foster broad awareness of the policy and practice connection in states (Thursday, 
August 3) 

• increase capacity to work more effectively with model developers and others 
providing services to schools (Friday, August 4) 

• expand knowledge and skills of external facilitation for comprehensive school reform 
(Friday, August 4) 

• provide an opportunity for states to reflect on Year 1 and plan for Year 2 (Saturday, 
August 5) 

• celebrate and acknowledge contributions of external facilitators (Saturday, August 

5) 
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Audience of and Purpose for This Report 

This report is intended for AEL CSRD staff involved in leading the Academy and the AEL 
Corporate Leadership Group. Others who might find this report of use include the Office of 
Educational Research and Improvement, from whom AEL receives funding, and other education 
researchers and practitioners interested in comprehensive school reform or external facilitation. 

The purpose of this report is to provide AEL CSRD staff with information about the strengths 
and weaknesses of the 2000 Academy institute so that they may make any needed changes in future 
Academy events. Specifically, participants were asked to complete evaluation forms about their 
satisfaction with various characteristics of several presentations and the institute overall. This report 
will also describe any changes between 1999 and 2000 in participants’ self-assessments of their 
external facilitation skills and the types of concerns that they have about their role. Finally, the 
degree to which the institute met its goals will be assessed. 
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METHODS 

Overall, the evaluation assessed participants’ satisfaction with various characteristics of four 
institute sessions, satisfaction with the entire institute, concerns about external facilitation as an 
innovation, and skills and knowledge associated with the role of external facilitator. To these ends, 
four types of questionnaires were administered during the 2000 institute, and are described below. 
It should be noted that although interviews and participant observation would have added depth and 
richness to the evaluation, such strategies were not considered cost effective during this year’s 
institute. 

Evaluation Instruments 

Stages of Concern Questionnaire. The Stages of Concern (SoC) Questionnaire (Hall, 
George, & Rutherford, 1 986) is an instrument that measures seven identified stages of concern that 
respondents experience as they progress through a change effort (see Appendix A). Developed by 
the Concems-Based Adoption Model (CBAM) researchers, the SoC questionnaire is a 35-item, 
paper-and-pencil survey instrument. Each of 35 stimulus statements solicits a response on a 7-point 
Likert scale. Responses along the 7-point scale indicate a respondent’s degree of concern about the 
stimulus statement. The lowest response option on the scale is (0) for irrelevant, while the highest 
response is (7) for very true of me now. The 35 stimulus statements are generic to any innovation 
and therefore can be applied to, in this case, the role of external facilitator. The text in the cover 
page of the instrument states the specific name of the innovation to which the generic stimulus 
statements refer and directs the respondents to keep only that innovation in mind when completing 
the instrument. The SoC Questionnaire was administered to participants at the beginning of the 1 999 
summer institute, and again at the end of the 2000 institute. 

External Facilitator Self-Assessment Instrument. This instrument was developed by AEL 
evaluators to assess the baseline skills and knowledge of the participants regarding the specific skills 
associated with the role of external facilitator (see Appendix B). The instrument was first 
administered at the beginning of the 1 999 summer institute; it was readministered at the close of the 
2000 event. 

Four Module Evaluation Instruments. F our instruments (see Appendix C) were developed 
by AEL evaluators to assess the extent to which participants were satisfied with characteristics of 
four sessions presented during the institute and the degree to which the sessions met their goals. Each 
instrument solicited responses to 1 7 items about sessions via a 5-point Likert-type scale, with anchor 
points of l=not at all and 5=very much. Participants were also requested to rate the level of 
knowledge they gained from each session and the extent to which they continued to need further 
information. Response options ranged from l=none or not at all to 5=very much or a lot. 

The Change-Adept Schools Instrument was specifically designed to measure 
participants’ opinions of the session on nurturing change-adept schools presented by 
Philip Schlechty 

The Dimensions of Change Instrument was specifically designed to measure 
participants’ opinions of the session on dimensions of change presented by Andy 
Hargreaves 
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• The Role of the District in School Reform Instrument was specifically designed to 
measure participants’ opinions of the session on the role of the local school district 
in school reform presented by Dale Kalkofen 

• The Facilitating Change in Schools Instrument was specifically designed to measure 
participants’ opinions of the session on facilitating change in schools presented by 
Ben Perez 

Overall Institute Evaluation Form. This instrument was developed by AEL evaluators to 
assess participants’ overall impressions of the summer institute and the degree which they thought 
institute goals and objectives had been met (see Appendix D). 

Review of Institute Materials. In addition to designing and analyzing evaluation 
instruments, the evaluator also reviewed the institute agenda and other materials distributed at the 
event for alignment with presentations, quality of the information, and utility to participants. The 
institute agenda is available in Appendix E. 

Data Collection and Data Analysis 

Instruments were administered in a group setting by AEL staff. The SoC Questionnaire and 
the self-assessment instruments were administered at the beginning of the institute. Session 
evaluation forms were administered immediately after the sessions were completed. The overall 
institute evaluation form was administered at the conclusion of the institute. 

All data analyses were conducted by AEL evaluators. Quantitative data analyses were 
completed using SPSS for Windows and a custom AEL analysis program for the SoC instrument. 
Descriptive statistics were calculated for each of the instruments. Tests of statistical significance and 
effect sizes were calculated for pre- and posttest administrations of the SoC and self-assessment 
instruments. Coefficient alpha (a measure of internal consistency reliability) was also calculated for 
each instrument. 
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FINDINGS 



Institute Participants 

Forty-seven participants attended the 2000 institute of the Academy for External facilitators. 
Attendance by state is as follows: Kentucky, 5; Tennessee, 14; Virginia, 19; and West Virginia, 9. 
Participants were individuals selected to serve as external facilitators in schools implementing CSRD 
reform and various state departments of education staff. In addition, 1 0 AEL staff were available to 
facilitate the event. 

Thirty-seven attendees completed a brief demographic questionnaire attached to the final 
evaluation form at the end of the institute. Nearly half of the respondents (48%) did not reply when 
asked to describe their primary role or affiliation. Twenty percent explicitly noted that they worked 
for a state department of education, and 1 7% described themselves as external facilitators. Three 
percent each responded that they were education consultants or staff of the Virginia Governor’s Best 
Practice Centers. The remaining 1 0% gave specific titles, but did not attribute these to any particular 
organizational affiliation. 

Thirty-seven attendees replied when asked about their full- or part-time employment status. 
Of these, about half (51%) worked full-time. Of the 23 respondents replying to a query about the 
grade levels they supervised or administered, 78% chose the option representing preK-12. PreK-8 
was selected by 4%, preK-6 by 13%, and the “other” option by 4%. Asked about the urbanicity or 
rurality of the schools in which they worked, more than half (54%) of the 35 respondents reported 
involvement with rural schools. Only 3% worked in suburban locales, while 1 1% were in urban 
schools. Slightly less than a third (31%) said they worked in more than one setting. 

The mean years respondents had worked in education was 29.75 (standard deviation [SD] 
of 7.49). Years in education ranged from 12 for one respondent to 44 for another. The majority of 
the 37 respondents to the demographic items were women (76%); the mean age of the 35 participants 
replying was 54 (SD 7.61), with ages ranging from 37 to 69. Thirty-seven respondents replied when 
asked for information about their education levels. Respondents were highly educated, with 97% 
having at least a master’s degree. Somewhat less than a third of the total respondents (27%) had 
earned a doctorate degree. 

Facilitating Change in Schools: What Does it Take? 

Thirty-seven participants completed the evaluation form about the session on facilitating 
change in schools. This instrument had a coefficient alpha of r = .91, indicating a high level of 
consistency among items. With mean ratings of satisfaction ranging from 4.94 to 4.62 on the 5-point 
scale, this session appears to have been quite well-received. The two highest mean ratings were given 
for the professional (4.94, SD 0.23) and appealing (4.92, SD 0.28) manner in which the session was 
presented and for the competence of the presenter (4.92, SD 0.28). The small standard deviations 
suggest that respondents were in close agreement about these characteristics. 

The extent to which the presentation increased communication and collegiality with others 
received the lowest mean rating (4.62, SD 0.72). Also less well rated was the degree to which the 
session was relevant to participants’ needs (4.67, SD 0.48). Nonetheless, these scores are very high 
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on the 5-point scale. Additional descriptive statistics for this section of the form are presented in 
Table 1. 



Table 1 

Mean Ratings of the Session Facilitating Change in Schools: What Does it Take? 



Item: Extent to which the session . . . 


N 


Mean 


SD 


1 . Had clear outcomes 


36 


4.69 


0.58 


2. Included clear directions for activities 


36 


4.69 


0.67 


3. Facilitated development of new skills 


37 


4.68 


0.53 


4. Was conducted in an appealing manner 


37 


4.92 


0.28 


5. Was conducted in a professional manner 


36 


4.94 


0.23 


6. Was conducted by competent presenter(s) 


37 


4.92 


0.28 


7. Had activities that were well sequenced 


37 


4.73 


0.45 


8. Had activities that reinforced content 


37 


4.73 


0.45 


9. Included appropriate examples 


37 


4.81 


0.46 


10. Was relevant to my needs 


36 


4.67 


0.48 


1 1 . Had meaningful involvement of participants 


37 


4.78 


0.53 


12. Caused me to reflect on my practices 


37 


4.78 


0.42 


13. Caused me to examine some of my attitudes 


37 


4.76 


0.44 


14. Stimulated me to want to use the concepts, skills, 
and/or materials presented 


37 


4.76 


0.44 


15. Provided materials that will be useful to me in my 
continuing work as an external facilitator 


37 


4.68 


0.48 


16. Increased communication and collegiality with 
others 


37 


4.62 


0.72 


17. Increased my understanding of what it takes to 
facilitate change in schools 


37 


4.70 


0.52 



Respondents were also asked to rate the amount of knowledge they had gained during the 
session about external facilitation and about external facilitation skills that could be added to their 
repertoire. Then, respondents were requested to rate their level of need for further information 
regarding the topics. 

Respondents indicated that they had gained much knowledge about external facilitation of 
CSRD (4.69, SD 0.53) and that they had expanded their repertoire of external facilitation skills (4.72, 
SD 0.5 1). However, respondents also reported continuing to have a moderate need for information 
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about both, with a mean of 3.69 (SD 1.00) for knowledge of external facilitation and a mean of 3.79 
(SD 0.94) for external facilitation skills. See Table 2 for descriptive statistics. 

Table 2 

Knowledge Gain and Further Need for Information about External Facilitation 



Item 


Knowledge Gain 


Further Need 


N 


Mean 


SD 


N 


Mean 


SD 


18. Expanded my knowledge of external 

facilitation for comprehensive school reform 


36 


4.69 


0.53 


29 


3.69 


1.00 


1 9. Expanded mv repertoire of external 

facilitation skills for comprehensive school 
reform 


36 


4.72 


0.51 


29 


3.79 


0.94 



Overall, this presentation was quite well-received by respondents. Even the lowest rated 
items were relatively high on the 5-point scale. Although participants reported having a moderate 
need for further information, this was not an overwhelming concern. 

Dimensions of Change in School Reform 

Forty-eight respondents replied to the evaluation form for the session about dimensions of 
change in school reform. (It is unclear why 48 evaluation forms were completed when only 47 
individuals were counted as participants.) This instrument had a coefficient alpha of r = .93, 
revealing a high level of consistency among items. The highest rating was given for the competence 
of the presenter, with a mean of 4.96 (SD 0.20). Also highly rated was the professional manner in 
which the session was presented (4.83, SD 0.38). Least well-rated was the extent to which the 
presentation increased communication and collegiality among participants, with a mean of 4.28 (SD 
0.78). With a mean of 4.43 (SD 0.74), the degree to which the session included meaningful 
involvement of participants was the second lowest rated item. See table 3 for further detail. 

Respondents were also asked to rate the amount of knowledge they had gained during the 
session about dimensions of change, particularly in terms of the degree to which the session 
expanded their understanding of how change is planned, managed, and implemented. Then, 
respondents were requested to rate their level of need for further information regarding the topic. 

Although respondents reported that they had gained much knowledge about how change is 
planned, managed, and implemented (4.55, SD 0.62), they nonetheless also reported a fairly 
significant need for further information (4.42, SD 0.63). Descriptive statistics are shown in Table 
4. 



Overall, this session was received very well by respondents, who thought it was presented 
in a professional manner by a competent presenter. Although the presentation did not necessarily 
engage participants in activities that enhanced communication and collegiality, this did not appear 
to lessen respondents’ regard for the session. Respondents learned much about dimensions of change 
but continued to have need for more information. 
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Table 3 

Mean Ratings of the Session School Reform: Understanding Dimensions of Change 



10 



Item: Extent to which the session . . . 


N 


Mean 


SD 


1 . Had clear outcomes 


48 


4.67 


0.56 


2. Included clear directions for activities 


47 


4.58 


0.58 


3. Facilitated development of new skills 


47 


4.53 


0.62 


4. Was conducted in an appealing maimer 


48 


4.75 


0.57 


5. Was conducted in a professional manner 


48 


4.83 


0.38 


6. Was conducted by competent presenter(s) 


48 


4.96 


0.20 


7. Had activities that were well sequenced 


48 


4.65 


0.57 


8. Had activities that reinforced content 


46 


4.52 


0.66 


9. Included appropriate examples 


48 


4.69 


0.51 


10. Was relevant to my needs 


48 


4.79 


0.41 


1 1 . Had meaningful involvement of participants 


47 


4.43 


0.74 


12. Caused me to reflect on my practices 


48 


4.75 


0.48 


13. Caused me to examine some of my attitudes 


48 


4.71 


0.50 


14. Stimulated me to want to use the concepts, skills, 
an^or materials presented 


48 


4.63 


0.67 


15. Provided materials that will be useful to me in my 
continuing work as an external facilitator 


47 


4.60 


0.74 


16. Increased communication and collegiality with 
others 


46 


4.28 


0.78 


17. Increased my understanding of the dimensions of 
change 


48 


4.75 


0.44 



Table 4 

Knowledge Gain and Further Need for Information about Dimensions of Change 



Item 


Knowledge Gain 


Further Need 


N 


Mean 


SD 


N 


Mean 


SD 


1 8 . Expanded my understanding of how change 
is planned, managed, and implemented 


47 


4.55 


0.62 


41 


4.42 


0.63 
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Creating Change-Adept Schools 

A total of 37 participants completed the evaluation form for the session about creating 
change-adept schools. With a coefficient alpha of r = .90, this instrument possesses sufficient 
internal reliability. Receiving the highest mean rating was the extent to which the session was 
conducted by competent presenters (4.89, SD 0.32). Also highly rated was the degree to which (1) 
the presentation increased respondents’ understanding of how to create change-adept schools (4.84 
(SD 0.44) and (2) caused respondents to reflect on their practice (4.84, SD 0.37). See Table 5 for 
further detail. 

The degree to which the session had meaningful involvement of participants, however, 
received the lowest mean rating of 3.97 (SD 1.30). The large standard deviation suggests that there 
was some disagreement among respondents about this issue. Receiving the second lowest mean 
rating was the extent to which the presentation encouraged communication and collegiality with 
other participants (4.03, SD 1.33). Again, the large standard deviation indicates that this view was 
not consistent among respondents. The extent to which the session included clear directions for 
activities had the largest standard deviation of 1.49, suggesting that directions were clear to some 
participants but not to others. 

Respondents were also asked to rate the amount of knowledge they had gained during the 
session about creating change-adept schools. Then, respondents were requested to rate their level of 
need for further information regarding the topic. 

Those completing the evaluation form indicated a large increase in their understanding of the 
topic, with a mean rating of 4.67 (SD 0.63) for their knowledge gain. Nonetheless, respondents also 
reported a fairly substantial need for further information about how to create schools adept at 
undertaking and sustaining change (4.55, SD 0.62). Descriptive statistics for this section of the 
instrument are presented in Table 6. 

Overall, as with other sessions, respondents reported that this session was quite satisfactory. 
They thought the presentation was given by a competent presenter and that their understanding of 
how to create change-adept schools had been increased. Nonetheless, respondents also reported a 
continuing need for information on the topic. 
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Table 5 

Mean Ratings for the Session Creating Change-Adept Schools 



12 



Item: Extent to which the session . . . 


N 


Mean 


SD 


1 . Had clear outcomes 


37 


4.60 


0.87 


2. Included clear directions for activities 


27 


4.07 


1.49 


3. Facilitated development of new skills 


35 


4.63 


0.77 


4. Was conducted in an appealing manner 


37 


4.70 


0.57 


5. Was conducted in a professional maimer 


37 


4.76 


0.55 


6. Was conducted by competent presenter(s) 


37 


4.89 


0.32 


7. Had activities that were well sequenced 


32 


4.28 


1.11 


8. Had activities that reinforced content 


31 


4.16 


1.29 


9. Included appropriate examples 


37 


4.62 


0.83 


10. Was relevant to my needs 


36 


4.72 


0.57 


1 1 . Had meaningful involvement of participants 


35 


3.97 


1.30 


12. Caused me to reflect on my practices 


37 


4.84 


0.37 


13. Caused me to examine some of my attitudes 


37 


4.73 


0.45 


14. Stimulated me to want to use the concepts, skills, 
and/or materials presented 


37 


4.76 


0.50 


15. Provided materials that will be useful to me in my 
continuing work as an external facilitator 


33 


4.09 


1.31 


16. Increased communication and collegiality with 
others 


33 


4.03 


1.33 


17. Increased my understanding of how to create 
change-adept schools 


37 


4.84 


0.44 



Table 6 

Knowledge Gain and Need for Further Information about Change-Adept Schools 



Item 


Knowledge Gain 


Further Need 


N 


Mean 


SD 


N 


Mean 


SD 


18. Gained knowledge about the creation of 
change-adept schools 


36 


4.67 


0.63 


33 


4.55 


0.62 
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The Role of the District in School Reform 

Fifteen respondents replied to the evaluation form for the session on the role of the school 
district in reform (see Table 7). Other sessions were scheduled at the same time as this session; the 
number of respondents is therefore smaller than the number for sessions scheduled alone. The 
evaluation instrument for this session had a coefficient alpha of r = .97, indicating very high 
consistency among items. Competence of the presenter (4.73, SD 0.46) and the professional manner 
in which the session was presented (4.67, SD 0.48) received the highest mean ratings. The lowest 
ratings were the extent to which the presentation had meaningful involvement of participants (4.00, 
SD 1.18) and had well-sequenced activities (4.00, SD 0.76). Some items had relatively large standard 
deviations, suggesting that respondents were not congruent in their appraisals of the session. 

Table 7 

Mean Ratings for the Session The Role of the District in School Reform 



Item: Extent to which the session . . . 


N 


Mean 


SD 


1 . Had clear outcomes 


15 


4.53 


0.64 


2. Included clear directions for activities 


13 


4.15 


0.80 


3. Facilitated development of new skills 


14 


4.21 


0.80 


4. Was conducted in an appealing manner 


15 


4.33 


0.72 


5. Was conducted in a professional manner 


15 


4.67 


048 


6. Was conducted by competent presenter(s) 


15 


4.73 


0.46 


7. Had activities that were well sequenced 


15 


4.00 


0.76 


8. Had activities that reinforced content 


15 


4.07 


1.10 


9. Included appropriate examples 


15 


4.53 


0.64 


10. Was relevant to my needs 


15 


4.40 


0.74 


1 1 . Had meaningful involvement of participants 


14 


4.00 


1.18 


12. Caused me to reflect on my practices 


15 


4.40 


1.06 


13. Caused me to examine some of my attitudes 


15 


4.13 


1.13 


14. Stimulated me to want to use the concepts, skills, 
and/or materials presented 


15 


4.13 


1.25 


1 5. Provided materials that will be useful to me in my 
continuing work as an external facilitator 


15 


4.40 


0.91 


1 6. Increased communication and collegiality with 
others 


14 


4.21 


1.18 


1 7. Increased my understanding of the role of the 
district in school reform 


15 


4.47 


0.83 



O 
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Table 8 

Knowledge Gain and Need for Further Information about 
the Role of the District in School Reform 



Item 


Knowledge Gain 


Further Need 


N 


Mean 


SD 


N 


Mean 


SD 


1 8. Enhanced mv knowledge of the role of the 
district in school reform 


15 


4.27 


0.88 


14 


3.93 


1.14 


19. Increased my capacity to work with district 
staff 


14 


4.21 


0.89 


13 


4.00 


1.16 



Respondents were also asked to rate the amount of knowledge they had gained during the 
session about the role of the district in school reform and about increasing their capacity to ^work 
with district staff. Then, respondents were requested to rate their level of need for further information 
regarding the topics. 

Participants reported gaining much knowledge about the role of the school district in reform 
(4.27, SD 0.88) and an enhanced capacity to work with district staff (4.21, SD 0.89). Yet they also 
indicated a need for further information about both issues, with means of 3.93 (SD 1.14) and 4.00 
(SD 1.16). The relatively large standard deviations suggest that respondents were not in complete 
agreement about their continuing need for such information, however. These statistics are shown in 
Table 8. 

In sum, this session was well-received by respondents. They perceived that the presentation 
was given by a competent presenter in a professional manner. And participants gained knowledge 
about the role of the school district in reform, and increased their capacity to work with school 
districts, although they had a moderate need for further information about the issues. 

External Facilitator Self-Assessment Instrument 

Participants were asked to rate their level of skill and knowledge with regard to 20 roles or 
areas of expertise associated with external facilitation. Participants used a scale of 1-10, with 1 
indicating limited skills and knowledge and 10 indicating exceptional skills and knowledge. The 
instrument had a coefficient alpha of r = .97, indicating high internal consistency among the items. 

Twenty-eight Academy participants completed the Self-Assessment Instrument at the 2000 
institute. Mean ratings for the 20 items ranged from 5.56 to 7.79 on the 10-point scale (SDs ranged 
from 1.31 to 2.61), suggesting that participants tended to think they possessed moderate levels of 
skill and knowledge about each area associated with the role of external facilitator. See Table 9 for 
further detail. 

Overall, respondents reported being most skillful and knowledgeable in the areas of 
collaborative leadership, the change process, school reform, decision making, listening, and 
communicating. Participants thought they possessed approximately average levels of skill and 
knowledge in terms of CSRDnet and technology, although these areas received the lowest mean 
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ratings. Diversity also appears to be an area of some concern for respondents, though this topic too 
received ratings above scale midpoint. 



Table 9 

2000 Self-Assessment Descriptive Statistics 



Items in Mean Rank Order 


N 


Mean 


SD 


1. CSRDnet 


27 


5.56 


2.61 


2. Technological abilities 


28 


5.96 


1.92 


3. Diversity 


28 


6.39 


1.69 


4. Reform models and strategies 


28 


6.71 


1.61 


5. Data interpretation 


28 


6.79 


1.83 


6. Environmental scanning 


27 


6.82 


1.86 


7. Network development 


28 


6.82 


1.92 


8. Standards, assessment, and testing 


28 


6.89 


1.83 


9. Resource identification 


28 


6.96 


1.55 


1 0. Group process 


28 


7.04 


1.64 


11. Formative evaluation 


28 


7.07 


1.80 


12. Data collection 


28 


7.11 


1.85 


13. Teambuilding 


28 


7.11 


1.60 


14. Situational analysis 


28 


7.18 


1.61 


15. Communications 


28 


7.36 


1.47 


16. Listening 


28 


7.43 


1.87 


1 7. Decision making 


28 


7.43 


1.35 


18. School reform 


28 


7.68 


1.31 


19. The change process 


28 


7.75 


1.60 


20. Collaborative leadership 


28 


7.79 


1.55 



Paired sample t tests were calculated to discern any statistically significant differences — ^that 
is, differences not likely due to chance or sampling error — in the skill and knowledge level of 
participants in attendance at both the 1999 and 2000 Academy institutes. In other words, t tests were 
run only on scores of those external facilitators who had participated in both the 1999 and 2000 
institutes (N=15); others were excluded from the analysis. 
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It should be noted that the assumptions of the t test were violated in this study. The sample 
was not random, nor was it assumed that the data were drawn from a normally distributed 
population. Phillips (1982) contends, however, that “since those assumptions now appear to be far 
less important than originally thought, the recent trend toward increasing use of distribution-free tests 
is currently being reversed” (p. 139). Likewise, Glass and Hopkins (1984) report research suggesting 
that violation of the assumptions of normality and homogeneity of variance has little impact upon 
the robustness of t tests. For these reasons, the evaluator chose to use the t test to explore the 
statistical significance of pre- and posttest differences on the Self-Assessment Instrument. 

Twelve statistically significant increases and no statistically significant decreases were 
located at the .05 level of significance. Increases were found in ratings of respondents’ levels of skill 
and knowledge about school reform, reform models and strategies, technological abilities, CSRDnet, 
team building, group process, collaborative leadership, resource identification, network development, 
the change process, environmental scanning, and situational analysis. T test statistics are presented 
in Table 10. 

Statistical significance alone, however, does not indicate the meaningfulness of findings; 
rather, it indicates the rareness of findings. The calculation of effect size allows the conversion of 
statistically significant results into the standard deviation metric, providing a better analysis of 
practical significance. Thus, effect sizea were calculated to estimate the practical significance of the 
pre- and posttest scores expressed in standard deviation units. 

Effect sizes for statistically significant increases were quite large, according to the 
conventions established by Cohen (1988). The smallest effect size was d = .49, representing a 
moderate increase in respondents’ levels of skill and knowledge with regard to collaborative 
leadership. The remaining effect sizes, ranging from d= .1% Xo d= were large, suggesting 
substantial growth in external facilitators’ perceptions of their skills and knowledge. 

To conclude, increases in respondents’ ratings of their skill and knowledge levels on 12 of 
the 20 items were likely not due to chance. Moreover, such increases were large. Growth in external 
facilitators’ knowledge and skills, then, is both substantial and not likely due to chance. Because 
such change is dramatic and statistically significant, it appears that participation in the External 
Facilitators Academy enhanced external facilitators’ skills and knowledge in important ways. 
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Table 10 

r Tests of 1999 and 2000 Administrations of the Self-Assessment Instrument 
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Self-Assessment Item 


Year 


N 


Mean 


SD 


t 


Sig. 


d 


School reform 


1999 


15 


6.40 


1.35 


4.000 


.001* 


.99 




2000 


15 


7.73 


.80 








Reform models and strategies 


1999 


15 


4.80 


2.08 


3.532 


.003* 


.93 


2000 


15 


6.73 


1.39 








Technological abilities 


1999 


15 


4.73 


1.79 


2.881 


.012* 


.78 


2000 


15 


6.13 


1.64 








CSRDnet 


1999 


15 


1.47 


1.13 


6.739 


.000* 


3.12 




2000 


15 


5.00 


2.07 








Team building 


1999 


15 


5.60 


1.55 


3.659 


.003* 


.95 


2000 


15 


7.07 


1.22 








Group process 


1999 


15 


5.80 


1.74 


3.400 


.004* 


.80 


2000 


15 


7.20 


1.32 








Diversity 


1999 


15 


5.53 


2.20 


1.023 


.324 


n/a 


2000 


15 


6.00 


1.31 








Communications 


1999 


15 ■ 


7.07 


1.62 


.299 


.769 


n/a 




2000 


15 


7.20 


.94 








Listening 


1999 


15 


7.40 


1.64 


.307 


.764 


nidi 


2000 


15 


7.53 


1.46 








Collaborative leadership 


1999 


15 


6.93 


1.62 


2.449 


.028* 


.49 


2000 


15 


7.73 


1.39 








Formative evaluation 


1999 


15 


6.13 


1.81 


1.407 


.181 


n/a 




2000 


15 


7.00 


1.41 








Data collection 


1999 


15 


6.80 


1.78 


.940 


.363 


n/a 




2000 


15 


7.27 


1.39 








Data interpretation 


1999 


15 


7.07 


1.79 


-.222 


.827 


n/a 


2000 


15 


6.93 


1.53 








Standards, assessment and testing 


1999 


15 


6.33 


1.45 


1.784 


.096 


n/a 


2000 


15 


7.00 


1.65 








Resource identification 


1999 


15 


5.00 


1.89 


4.740 


.000* 


1.02 




2000 


15 


6.93 


.96 








Network development 


1999 


15 


4.60 


2.61 


2.673 


.018* 


.82 


2000 


15 


6.73 


1.67 








The change process 


1999 


15 


6.20 


1.42 


5.000 


.000* 


1.18 


2000 


15 


7.87 


1.06 








Environmental scanning 


1999 


14 


4.14 


2.63 


3.631 


.003* 


.92 


2000 


14 


6.57 


1.02 








Situational analysis 


1999 


15 


4.07 


2.05 


5.890 


.000* 


1.49 


2000 


15 


7.13 


1.30 








Decision making 


1999 


14 


6.79 


2.19 


1.422 


.179 


n/a 


2000 


14 


7.43 


.94 









^ * Statistically significant at the .05 level. 

ERIC 
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Stages of Concern Questionnaire 

The Stages of Concern (SoC) Questionnaire was given near the end of the institute. 
According to the instrument developers, a concern is “the composite representation of the feelings, 
preoccupation, thought, and consideration given to a particular issue or task” (Hall, George, & 
Rutherford, 1977, p. 5). SoC data are intended for use by personnel involved in the adoption of an 
innovation for diagnostic purposes. As with many instruments, interpretations of SoC data should 
be treated as hypotheses and confirmed with respondents to maximize the usefulness of the 
questionnaire. 

Three types of analyses were conducted of the SoC data. First, an overall group profile was 
obtained for those completing the instrument at the 2000 institute. Second, percentile scores for 
each participant were obtained, and a dual peak analysis was conducted. Third, individual profiles 
for each participant were generated, and paired sample t tests were conducted to determine the 
presence of any statistically significant changes in external facilitators’ concerns between 1999 and 
2000. A total of 37 participants in the 2000 institute completed the survey. 

Overall Group Profile. The profile interpretations presented below are guided by the SoC 
technical manual (Hall, George, & Rutherford, 1977). As seen in Figure 1, the highest peak of 
concern in 2000 was at the Awareness level (Stage 0). Although Stage 0 scores are not generally 
interpreted in a group profile, the instrument developers contend that, for users of an innovation, 
scores above the bO* percentile suggest two alternative phenomena: respondents are either 
unconcerned with the innovation, focusing their attention on other work, or they have achieved a 
high level of comfort with it. External facilitators may also continue to struggle with the meaning 
of their work, as CSRD is as yet a relatively new program and their role an evolving one. Moreover, 
respondents perhaps continue to require a definition of the significance of the Academy to their 
work. 



Figure 1 

Stages of Concern Mean Percentiles, 1999-2000 




1999 2000 
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The second highest peak for the 2000 group is at Stage 5, Collaboration. This suggests that 
respondents are either genuinely interested in working together with others in an effort to enhance the 
innovation or are more interested in receiving ideas for their work from others. 

It is also notable that the lowest level of concern was at Stage 4, Consequence. This suggests 
that respondents have the least amount of concern with the consequences, or outcomes of, the 
innovation. Also of interest is that 1999 and 2000 group profiles do not appear significantly different. 
Although external facilitators are less concerned overall in 2000, the group profiles for both 
administration years reveal strikingly similar patterns of concern. Put another way, the group has not 
progressed through any of the developmental stages of concern, but overall their levels of concern are 
less intense. 

Dual Peak Analysis. The first and second highest percentile scores were obtained from each 
participant’s profile. This is one type of analysis suggested by the instrument’s developers. 

As noted earlier. Stage 0 (Awareness) scores are conventionally not included in analyses of 
the SoC when administered to users of an innovation. They are nonetheless presented in this report 
because, although Hall, George, and Rutherford (1977) maintain that high Stage 0 scores are 
generally accompanied by low Stage 1 and 2 scores, this is not the case for the 2000 SoC respondents. 
In addition, it may be important not to overlook the preponderance of Stage 0 peaks. 

Across all individual profiles, the most frequent high peak was at Stage 0. Eighteen of the 37 
respondents were most concerned with their awareness of the Academy (of these, one had an identical 
high score for Stage 1 , Information). This corroborates the findings presented earlier that participants 
continued to have relatively high needs for information about the topics covered in the four sessions 
that were evaluated. The second most frequent high peak was at Stage 5 (Collaboration), with 14 
respondents reporting the greatest amount of concern with collaborating with others (one of these had 
an identical high score for Stage 1). 

The most frequent second highest score was at Stage 0, with 12 respondents feeling some 
concern about their awareness of the innovation. For 1 1 respondents. Stage 1, Information, was the 
second highest area of concern. 

Stage 0 received either the first or second highest score from 29 of the 37 respondents 
(including one individual with identical peaks). Stage 5, Collaboration, received either the first or 
second highest score from 22 respondents (including one individual with identically scored peaks). 

Combinations of first and second high scores were also analyzed. The most frequent 
combination was of Stage 0 (Awareness) and Stage 5 (Collaboration). The next most frequent 
combination was of Stage 0 and Stage 1 (Information). 

In sum, it appears that many external facilitators continue to be focused on their awareness 
of the role of the external facilitator. In addition, many also appear concerned with collaborating with 
others; it is unclear, however, whether such concern is to improve the innovation and its effectiveness 
or to gain new ideas and strategies from others. 
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T Tests of 1999 and 2000 Scores. Paired sample t tests were conducted to determine if 
changes in levels of concern between the 1 999 and 2000 administrations of the SoC instrument were 
statistically significant (see Table 1 1). Decreases in mean raw scores for five stages of concern were 
found to be statistically significant at the .05 level. Hence, the diminishment in respondents’ concern 
about awareness, information, personal, management, and consequence issues is likely not due to 
chance alone. 

Cohen’s d was also calculated to determine the practical meaningfulness of statistically 
significant findings. With effect sizes ranging from d= 1.26 to 1.73, decreases in respondents’ 
levels of concern about awareness, information, personal, and management issues were substantial. 
However, with d = 0.35, decreases in scores for Stage 4 (Consequence) were small. 

In sum, it appears that external facilitators who had participated in the 1999 and 2000 
institutes were significantly and substantially less concerned about their awareness of the innovation, 
information levels, personal issues, and management considerations, and significantly and somewhat 
less concerned about consequences. Their levels of concern about collaboration and refocusing 
remained fairly stable in 1999 and 2000. 



Table 11 

r Tests of 1999 and 2000 Administrations of the Stages of Concerns Questionnaire 



Stage of Concern 


Year 


N 


Mean 


SD 


t 


Sig. 


d 


Stage 0 (Awareness) 


1999 


21 


16.19 


5.35 


3.686 


O 

O 

* 


1.26 




2000 


21 


9.43 


4.85 








Stage 1 (Information) 


1999 


21 


22.71 


7.38 


5.798 


.000* 


1.45 




2000 


21 


12.00 


7.29 








Stage 2 (Personal) 


1999 


21 


20.29 


5.56 


4.990 


.000* 


1.73 




2000 


21 


10.67 


7.70 








Stage 3 (Management) 


1999 


37 


41.51 


23.61 


10.395 


.000* 


\21 




2000 


37 


11.41 


6.02 








Stage 4 (Consequence) 


1999 


37 


20.92 


20.61 


3.426 


.002* 


0.35 




2000 


37 


13.70 


8.92 








Stage 5 (Collaboration) 


1999 


21 


26.19 


8.90 


1.758 


.094 


n/a 




2000 


21 


22.67 


10.70 








Stage 6 (Refocusing) 


1999 


21 


16.71 


7.73 


0.541 


.594 


n/a 




2000 


21 


15.67 


8.39 









* Statistically significant at the .05 level. 



Overall Institute Evaluation 

Thirty-five participants completed and returned the final evaluation form concerning the entire 
institute. Section A of the evaluation form asked respondents to reply to items related to various 
logistical aspects of the institute. Section B asked respondents to rate the extent to which various 
institute objectives were met. Anchor points for items in these two sections ranged from l=never or 
not at all to 5=always or very. Finally, Section C provided three open-ended prompts for respondents 
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to complete concerning the high point of the event, suggestions for improvement, and any remaining 
concerns. This instrument possesses relatively high internal consistency among quantitative items, 
with a coefficient alpha of r = .86. 

As presented in Table 12, mean ratings in response to items in Section A of the evaluation 
form were quite high on the 5-point scale, ranging from a low of 4.43 (SD 0.74) in terms of the extent 
to which AEL carried out plaimed activities at their scheduled times to a high of 4.97 (SD 0. 1 7) with 
regard to the responsiveness of AEL staff and presenters to requests for service or assistance. These 
mean ratings, along with the relatively small standard deviations, suggest that respondents tended to 
be in agreement about their high level of satisfaction with the knowledge, skills, and responsiveness 
of AEL staff; the usefulness of materials and the overall institute to their ongoing work; and the 
location, format, and meeting facilities of the institute. 

Table 12 

Mean Ratings of Overall Institute, Section A 



Item 


N 


Mean 


SD 


1. Did AEL carry out planned activities at the times scheduled? 


35 


4.43 


0.74 


2. How responsive were AEL staff and/or presenters to your requests for 
service and/or assistance during the Institute? 


35 


4.97 


0.17 


3. How knowledgeable were AEL staff and/or presenters? 


35 


4.91 


0.28 


4. How skilled were AEL staff and/or presenters in completing their 
tasks? 


35 


4.91 


0.28 


5. How well were your information needs met at this Institute? 


35 


4.74 


0.51 


6. How useful were the materials provided to you during this Institute? 


35 


4.77 


0.43 


7. How satisfied were you with the location of this event? 


35 


4.91 


0.28 


8. How satisfied were you with the format of this Institute? 


35 


4.60 


0.60 


9. How satisfied were you with the meeting facilities at this event? 


35 


4.83 


0.51 


10. How useful do you anticipate this Institute will be to your work? 


35 


4.83 


0.38 



Mean ratings in reply to Section B items were similarly high on the 5-point scale (see Table 
13). The highest mean rating was given in response to the item concerning respondents’ broadened 
understanding of school reform and the change process (4.83, SD 0.38). Also highly rated was the 
extent to which the institute increased respondents’ repertoire of strategies to help schools build their 
capacity for continuous improvement, with a mean of 4.77 (SD 0.42). 

Receiving the lowest mean rating was the extent to which respondents informed their states’ 
future reform efforts through sharing lessons learned (4.00, SD 0.90). The item receiving the next 
lowest rating of 4.17 (SD 0.87) concerned the usefulness of the opportunity for respondents to plan 
for Year 2 with their state colleagues. 
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In sum, mean ratings on this section of the overall evaluation form indicate that institute 
objectives were met well. Even the lowest rated item is relatively high on the 5-point scale. 



Table 13 

Mean Ratings of Overall Institute, Section B 



Item 


N 


Mean 


SD 


1 . To what extent did the Institute broaden your understanding of school 
reform and the change process? 


35 


4.83 


0.38 


2. To what extent did you increase your repertoire of strategies for 
helping schools build their capacity for continuous improvement? 


35 


4.77 


0.42 


3. To what extent did you assist to inform your state’s future reform 
efforts through sharing lessons learned? 


33 


4.00 


0.90 


4. To what extent were you able to expand your resource network? 


33 


4.58 


0.56 


5. To what extent did you acquire an expanded understanding of how 
change is planned, managed, and implemented? 


33 


4.58 


0.61 


6. To what extent was your awareness of the connection between policy 
and practice in states broadened? 


33 


4.24 


0.71 


7. To what extent did you develop new collegial relationships? 


33 


4.49 


0.67 


8. To what extent did you learn from other participants? 


33 


4.46 


0.62 


9. To what extent do you feel better prepared to work effectively with 
model developers and others providing services to schools? 


33 


4.27 


0.67 


10. To what extent were your external facilitation skills augmented? 


33 


4.64 


0.60 ■ 


1 1 . To what extent was your knowledge of external facilitation enhanced? 


33 


4.58 


0.61 


12. To what extent was the opportunity to reflect on Year 1 with your state 
colleagues useful? 


30 


4.40 


0.93 


13. To what extent was the opportunity to plan for Year 2 with your state 
colleagues useful? 


30 


4.17 


0.87 



Twenty-nine of the 35 respondents to the evaluation form replied to the first open ended 
question soliciting their perceptions of the institute’ s high point. Three of these respondents provided 
answers with two themes. 



Twenty-two respondents reported that the presenters had been the high point of the institute. 
Fifteen of these mentioned at least one presenter by name. The networking and discussions were the 
high point, according to four respondents, and the useful materials to two others. Four replies were 
idiosyncratic. One reported that the high point had been “applying knowledge at my school.” Another 
thought that the opportunity to present work to the state school superintendents had been most 
valuable. One simply said that there were “lots of high points.” And another made two comments 
either overheard at the institute or that were relevant to the question posed: “Comment — a child has 
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just one chance at 2"'* grade. Another comment — We are experimenting with the very neediest, poor 
children who have no other resources.” 

Eighteen respondents replied to the prompt for suggestions to improve the institute. Time was 
the focus of eight respondents’ answers. Of these, four requested more time with the presenters; 
another requested more time for state planning. Two respondents reported that they had no 
suggestions for improvement, and another two gave positive feedback about the event. Two 
respondents suggested that the institute only be scheduled during weekdays rather than on Saturday. 
Four replies were unique. One respondent suggested greater focus, requesting a “day long focus on 
[a] single concept.” Another suggested shortening presentations of state stories, while, similarly, 
another wrote, “less on our presentation.” “Friday’s format could be reviewed,” wrote yet another 
respondent. 

S ixteen of the 3 5 evaluation form respondents answered the final open ended question seeking 
their remaining concerns. Six respondents replied that they had no concerns to report, two of whom 
also praised AEL and the Academy. Three respondents were concerned that they had so much yet to 
learn about external facilitation and school reform; one of these participants also requested that 
Academy participation be extended to a third year. The remaining replies were idiosyncratic and are 
presented in Table 14. 



Table 14 

Remaining Responses to Question C3 



Idiosyncratic Responses to Question C3 

“How to fit EF [external facilitation] into my job.” 

“Future.” 

“Facilitating change in schools without being an advocate.” 

“Just more of the innovative, on-the-edge learning.” 

“Preparing schools to sustain change.” 

“Can I transfer all of this into successful outcomes for students? Really help charge my profession into 
a caring, competent workforce for America’s kids?” 

“I think we need [to] work more together.” 




30 



24 



CONCLUSIONS AND RECOMMENDATIONS 



Conclusions 

Several conclusions may be drawn from the data collected at the 2000 summer institute of the 
Academy for External Facilitators. Comparative data also yield a few conclusions. 

The four individual sessions that participants were asked to evaluate were very well-received. 
Presenters were competent, and their presentations were professional and appealing. Although these 
sessions did not necessarily engage participants actively in communicating with their colleagues or 
involve them meaningfully, this was not of serious concern to evaluation form respondents. 

Taken as a whole, the 2000 institute provided useful information and materials to attendees, 
with responsive and knowledgeable assistance from AEL staff and presenters. In addition, various 
logistics of the event were satisfactory to most evaluation form respondents. 

Moreover, institute objectives were met well. Participants completed the institute with a 
broadened understanding of school reform and the change process, as well as an increased repertoire 
of external facilitation strategies. Although planning objectives were somewhat less well achieved 
than others, participants were not overly concerned with this. 

Attendees were very pleased with the presentations given by renowned education consultants 
and speakers. In fact, several suggested that the institute might have been improved had more time 
been allotted them. Networking opportunities were also useful to several attendees. 

More time could have been provided for states to plan their future reform work. But, overall, 
attendees were quite pleased with the institute and reported very little need for improvement of the 
event. 



A few participants were concerned about the how much learning they felt they still needed in 
order to be successful external facilitators. Several attendees had lingering concerns about the efficacy 
of their work and the challenges they faced. 

External facilitators possessed the most skills and knowledge in terms of collaborative 
leadership, the change process, school reform, decision making, listening, and communicating. 
Overall, participants felt their skill and knowledge levels were moderately high in these areas. They 
were least skilled and knowledgeable with regard to CSRDnet and technology. 



External facilitators participating in both the 1999 and 2000 institutes were significantly and 
substantially more skilled and knowledgeable in 2000 with regard to school reform, reform models 
and strategies, technological abilities, CSRDnet, team building, group process, resource identification, 
network development, the change process, environmental scanning, and situational analysis. And to 
a lesser extent, external facilitators were also more skilled and knowledgeable about collaborative 
leadership. 



Overall, attendees at the 2000 institute were less concerned about their roles as external 
facilitators than they had been the previous year. Although they remained most concerned about their 
q” wareness of the innovation and collaboration wdth others, they also continued to have a high concern 
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for information about external facilitation. External facilitators were least concerned about the 
consequences or outcomes of their work, as they had been the previous year. Individuals were most 
concerned with either awareness and information about the innovation, or awareness and 
collaboration with others. 

External facilitators who had participated in the 1999 and 2000 institutes were significantly 
and substantially less concerned about their awareness of the innovation, information levels, personal 
issues, and management considerations than they had been in 1999. Their levels of concern about 
collaboration and refocusing remained fairly stable in 1999 and 2000. 

In sum, external facilitators found the 2000 institute useful and pleasant to attend. Moreover, 
participants have strengthened external facilitation skills and have less intense concern about their 
roles as external facilitators 

Recommendations 

Recommendations for future External Facilitator Academy institutes are made below based 
upon the data collected during the 2000 summer event. 

AEL staff should continue to offer external facilitators the opportunity to hear significant 
presentations on school reform. However, AEL staff should take participants’ suggestions for 
improvement into account when developing future institutes. Such suggestions include scheduling 
institutes during weekdays, providing more time for participants to listen to and speak with presenters, 
and providing more time for state planning. 

Because external facilitation requires so many skills and much knowledge about school 
change, AEL could provide external facilitators with ongoing access to Academy resources and 
events. Mechanisms to ensure this have already been established in some instances, including 
additional Academy meetings and access to CSRDnet. 

Although external facilitators’ skills and knowledge in many areas have developed 
substantially during their participation in the Academy, others have not. AEL staff should focus their 
efforts in future Academy events on enriching external facilitators’ skills and knowledge about, for 
instance, data interpretation or diversity. In addition, participants continue to have the least amount 
of efficacy with regard to technology and CSRDnet. AEL staff might consider offering additional 
training, information, or support in these areas. 

To conclude, CSRD staff should continue to offer high quality information, materials, 
presentations, and other support to external facilitators as they continue their reform work. The 
success of the 2000 institute suggests that participants have appreciated the Academy, and more 
importantly, found it of use to their undertakings. 
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APPENDIX A: 



Stages of Concern (SoC) Questionnaire 



Concerns Questionnaire 



Name (Optional) 
Date Completed 



It is very important for continuity in processing the data associated with your participation in the AEL 
CSRD Academy for External Facilitators that we have a unique number that you can remember. Please 
record your Academy ID number: 



The purpose of this questionnaire is to determine what people who are using or thinking about using various 
programs are concerned about at various times during the innovation adoption process. The items were 
developed from typical responses of school and college teachers who ranged from "no knowledge at all" about 
various programs to "many years experience" in using them. Therefore, a good part of the items on this 
questionnaire appear to be of little relevance or irrelevant to you at this time. For the completely irrelevant 
items, please circle "0" on the scale. Other items will represent those concerns you do have, in varying degrees 
of intensity, and should be marked higher on the scale, according to the explanation at the top of each of the 
following pages. 



For example: 



This statement is very true of me at this time. 



This statement is somewhat true of me now. 



This statement is not at all true of me at this time. 



This statement is irrelevant to me. 



0 1 2 3 4 5 6© 
012 30567 
002 3 4 5 6 7 
© 1 23 4567 



Please respond to the item in terms of your present concerns, or how you feel about your involvement or 
potential involvement with the AEL CSRD Academy for External Facilitators. We do not hold to any one 
definition of this program, so please think of it in terms of your own perceptions of what it involves. Since this 
questionnaire is used for a variety of innovations, the name of the Academy never appears. However, phrases 
such as "the innovation," "this approach," and "the new system" all refer to the AEL CSRD Academy for External 
Facilitators. Remember to respond to each item in terms of your present concerns about your involvement or 
potential involvement with the AEL CSRD Academy for External Facilitators. 



Thank you for taking time to complete this task. 



Copyright © 1989 

Concerns Based Systems International 
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0 



1 



SCALE 

— 4 



Irrelevant Not true of me now 

1. I am concerned about students’ attitudes 
toward this innovation. 



Somewhat true of me now 



— 6 7 

Very true of me now 



2. I now know of some other approaches that 
might work better. 

3. I don’t even know what the innovation is. 



4 

4 



4. I am concerned about not having enough time 
to organize myself each day. 

5. I would like to help other faculty in their use 
of the innovation. 

6. I have a very limited knowledge about the innovation. 

7. I would like to know the effect of reorganization 
on my professional status. 

8. lam concerned about conflict between my 
interests and my responsibilities. 

9. I am concerned about revising my use of the innovation. 

10. I would like to develop working relationships 
with both our faculty and outside faculty 
using this innovation. 

11. lam concerned about how the innovation 
will affect students. 

12. lam not concerned about this innovation. 



13. I would like to know who will make the decisions 
in the new system. 

14. I would like to discuss the possibility of using 
the innovation. 



15. I would like to know what resources are available 
if we decide to adopt this innovation. 

16. lam concerned about my inability to manage 
all that the innovation requires. 

17. I would like to know how my teaching or 
administration is supposed to change. 

18. I would like to familiarize other departments or 
persons with the progress of this new approach. 



0 



1 



C-2 

Please continue on the next page. 
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19. 

20 . 

21 . 

22 . 

23. 

24. 

25. 

26. 

27. 

28. 

29. 

30. 

31. 

32. 

33. 

34. 

35. 



0 1 2 - 

Irrelevant Not true of me now 



SCALE 

3 4 5 6 7 

Somewhat true of me now Very true of me now 



I am concerned about evaluating my impact 
on students. 



0 1.2 



3 4 5 6 



I would like to help others in facilitating the use 
of this innovation. 



0 1 2 3 4 5 6 



I am completely occupied with other things. 

I would like to modify our use of the innovation 
based on the experiences of our students. 

Although I don’t know about this innovation, I am 
concerned about things in the area. 

I would like to excite my students about their part 
in this approach. 

I am concerned about time spent working with 
nonacademic problems related to this innovation. 

I would like to know what the use of the innovation 
will require in the immediate future. 

I would like to coordinate my efforts with others 
to maximize the innovation’s effects. 



0 



2 3 4 5 6 



0 1 2 3 4 5 6 

0 1 2 3 4 5 6 



0 1 2 3 4 5 6 



0 



2 3 4 5 6 



0 



2 3 4 5 6 



0 1 2 3 4 5 6 



I would like to have more information on time and 
energy commitments required by this innovation. 

I would like to know what other faculty are doing 
in this area. 



0 1 2 3 4 5 6 

0 1 2 3 4 5 6 



At this time, I am interested in learning about the 
innovation. 



0 1 2 3 4 5 6 



I would like to determine how to supplement, 
enhance or replace the innovation. 



0 



2 3 4 5 6 



I would like to use feedback from students to 
change the program. 

I would like to know how my role will change 
when I am using the innovation. 



0 1 2 3 4 5 6 

0 1 2 3 4 5 6 



Coordination of tasks and people is taking too 
much of my time. 

I would like to know how this information is 
better than what we have now. 



0 1 2 3 4 5 6 

0 1 2 3 4 5 6 



7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 
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AEL CSRD Academy for External Facilitators 
2000 Summer Institute 
Self-Assessment Instrument 



Please take a few moments to respond to the following items. Your replies will remain anonymous and 
confidential, so feel free to answer candidly. 



Please rate your current skills/knowledge level related to . . 

Limited skills 



Exceptional skills 



l. 


& knowledge 

School reform 1 2 


3 


4 


5 


6 


7 


8 


9 


& knowledge 
10 


2. 


Reform models and strategies 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


3. 


Technological abilities 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


4. 


CSRDnet 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


5. 


Team building 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


6. 


Group process 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


1. 


Diversity 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


8. 


Communications 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


9. 


Listening 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


10. 


Collaborative leadership 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


11. 


Formative evaluation 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


12. 


Data collection 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


13. 


Data interpretation 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


14. 


Standards, assessment and testing 1 2 


3 


4 


5 


6 


7 


8 


9 


10 


15. 


Resource identification 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


16. 


Network development 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


17. 


The change process 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


18. 


Environmental scanning 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


19. 


Situational analysis 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 


20. 


Decision making 


1 2 


3 


4 


5 


6 


7 


8 


9 


10 



Thank you! 
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APPENDIX C: 



Session Evaluation Forms 







Academy ID #: 

AEL CSRD Academy for ExtersaJ Eadlitators 
2000 Summer Institute 
Module Evaluation Form: 

Facilitating Change in Schools: What Does it Take? 
Presenter: Ben Perez 



fVe are interested in continuously improving our services. Please help us to do so. by taking a few 
moments to respond to the following items. Your replies will remain anonymous and confidential, so feel 
free to answer candidly. Thank you! 



Section A: Circle the number that best indicates the extent to which the session(s) on facilitating 
change in schools . . . 



1. 


Had clear outcomes 


Not at . 
all 

1 


2 


Neutral 

3 


4 


Very 

much 

5 


2. 


Included clear directions for activities 


1 


2 


3 


4 


5 


3. 


Facilitated development of new skills 


1 


2 


3 


4 


5 


4. 


Was conducted in an appealing manner 


1 


2 


3 


4 


5 


5. 


Was conducted in a professional manner 


1 


2 


3 


4 


5 


6. 


Was conducted by competent presenter(s) 


1 


2 


3 


4 


5 


7. 


Had activities that were well sequenced 


1 


2 


3 


4 


5 


8. 


Had activities that reinforced the content . 


1 


2 


3 


4 


5 


9. 


Included appropriate examples 


1 


2 


3 


4 


5 


10. 


Was relevant to my needs 


1 


2 


3 


4 


5 


11. 


Had meaningful involvement of participants 


1 


2 


3 


4 


5 


12. 


Caused me to reflect on my practices 


1 


2 


3 


4 


5 


13. 


Caused me to examine some of my attitudes 


1 


2 


3 


4 


5 


14. 


Stimulated me to want to use the concepts, 
skills, and/or materials presented 


1 


2 


3 


4 


5 


15. 


Provided materials that will be useful to me in 
my continuing work as an external facilitator 


1 


2 


3 


4 


5 


16. 


Increased communication and collegiality 
with others 


1 


2 


3 


4 


5 


17. 


Increased my understanding of what it takes 
to facilitate change in schools 


1 


2 


3 


4 


5 



Section B: Listed below are objectives for the session on facilitating change. You are asked to rate 
each objective on two dimensions: 1) knowledge you gained during the session, and 2) your need for 
further information regarding the objective. Scale : 1 = None or not at all to 5 = Very much or a lot 



18. 

19. 




Knowledge gain 

Expanded my knowledge of external 

facilitation for comprehensive school reform 1 2 3 4 5 

Expanded my repertoire of external 
facilitation skills for comprehensive school 
reform 1 2 3 4 5 



Further need 
1 2 3 4 5 

1 2 3 4 5 



Academy ID #: 

AEL CSRD Academy for External Facilitators 
2000 Summer Institute 
Module Evaluation Form: 

School Reform: Understanding Dimensions of Change 
Presenter: Andrew Hargreaves 



We are interested in continuously improving our services. Please help us to do so by taking a few 
moments to respond to the following items. Your replies will remain anonymous and confideniiai, so feel 
free to answer candidly. Thank you! 



Section A: Circle the number that best indicates the extent to which the session(s) on understanding 
dimensions of change . . . 



1. 


Had clear outcomes 


Not at 
all 

1 


2 


Neutral 

3 


4 


Very 

much 

5 


2. 


Included clear directions for activities 


1 


2 


3 


4 


5 


3. 


Facilitated development of new skills 


1 


2 


3 


4 


5 


4. 


Was conducted in an appealing manner 


1 


2 


3 


4 


5 


5. 


Was conducted in a professional manner 


1 


2 


3 


4 


5 


6. 


Was conducted by competent presenter(s) 


1 


2 


3 


4 


5 


7. 


Had activities that were well sequenced 


1 


2 


3 


4 


5 


8. 


Had activities that reinforced the content 


1 


2 


3 


4 


5 


9. 


Included appropriate examples 


1 


2 


3 


4 


5 


10. 


Was relevant to my needs 


1 


2 


3 


4 


5 


11. 


Had meaningful involvement of participants 


1 


2 


3 


4 


5 


12. 


Caused me to reflect on my practices 


1 


2 


3 


4 


5 


13. 


Caused me to examine some of my attitudes 


1 


2 


3 


4 


5 


14. 


Stimulated me to want to use the concepts, 
skills, and/or materials presented 


1 


2 


3 


4 


5 


15. 


Provided materials that will be useful to me in 
my continuing work as an external facilitator 


1 


2 


3 


4 


5 


16. 


Increased communication and collegiality 
with others 


1 


2 


3 


4 


5 


17. 


Increased my understanding of the dimensions 
of change 


1 


2 


3 


4 


5 



Section B: Listed below is the objective for the session on understanding dimensions of change. You 
are asked to rate the objective on two scales; 1) knowledge you gained during the session, and 2) your 
need for further information regarding the objective. 

Scale : 1 = None or not at all to 5 = Very much or a lot 



1 8. Expanded my understanding of how change 

is planned, managed and implemented 



Knowledge gain 
1 2 3 4 5 



Further need 
1 2 3 4 5 
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Academy ID #: 



AEL CSRD Academy for External Facilitators 
2000 Summer Institute 
Module Evaluation Form: . 

Creating Change-Adept Schools 
Presenter: Phil Schlechty 

We are interested in continuously improving our ser\’ices. Please kelp us to do so by taking a few 
moments to respond to the following items. Your replies will remain anonymous and confidential, so feel 
free to answer candidly. Thank you! 

Section A: Circle the number that best indicates the extent to which the session on creating change- 
adept schools . . . 







Not at 
all 


Neutral 




Very 

much 


1. 


Had clear outcomes 


1 2 


3 


4 


5 


2. 


Included clear directions for activities 


1 2 


. 3 


4 


5 


3. 


Facilitated development of new skills 


1 2 


3 


4 


5 


4. 


Was conducted in an appealing manner 


1 2 


3 


4 


5 


5. 


Was conducted in a professional manner 


1 2 


3 


4 


5 


6. 


Was conducted by competent presenter(s) 


1 2 


3 


4 


5 


7. 


Had activities that were well sequenced 


1 2 


3 


4 


5 


8. 


Had activities that reinforced the content 


1 2 


3 


4 


5 


9. 


Included appropriate examples 


1 2 


3 


4 


5 


10. 


Was relevant to my needs 


1 2 


3 


4 


5 


11. 


Had meaningful involvement of participants 


1 2 


3 


4 


5 


12. 


Caused me to reflect on my practices 


1 2 


3 


4 


5 


13. 


Caused me to examine some of my attitudes 


1 2 


3 


4 


5 


14. 


Stimulated me to want to use the concepts, 
skills, and/or materials presented 


1 2 


3 


4 


5 


15. 


Provided materials that will be useful to me in 
my continuing work as an external facilitator 


1 2 


3 


4 


5 


16. 


Increased communication and collegiality 
with others 


1 2 


3 


4 


5 


17. 


Increased my understanding of how to create 
change-adept schools 


1 2 


3 


4 


5 



Section B: Listed below is the objective for the session with model developers. You are asked to rate 
the objective on two dimensions: 1) knowledge you gained during the session, and 2) your need for 
further information regarding the objective. Scale : 1 = None or not at all to 5 = Very much or a lot 

Knowledge gain Further need 

1 8. Gained knowledge about the creation 

of change-adept schools 12345 12345 
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Academy ID #: 

AEL CSRD Academy for External Facilitators 
2000 Summer Institute 
Module Evaluation Form: 

The Role of the District in School Reform 
Presenter: Dale Kalkofen 

We are interested in continuously improving our services. Please help us to do so by taking a few 
moments to respond to the following items. Your replies will remain anonymous and confidential, so feel 
free to answer candidly. Thank you! 

Section A: Circle the number that best indicates the extent to which the session(s) on the role of the 
district in school reform ... 



1. 


Had clear outcomes 


Not at 
all 

1 


2 


Neutral 

3 


4 


Very 

much 

5 


2. 


Included clear directions for activities 


1 


2 


3 


4 


5 


3. 


Facilitated development of new skills . 


1 


2 


3 


4 


5 


4. 


Was conducted in an appealing manner 


1 


2 


3 


4 


5 


5. 


Was conducted in a professional manner 


1 


2 


3 


4 


5 


6. 


Was conducted by competent presenter(s) 


1 


2 


3 


4 


5 


7. 


Had activities that were well sequenced 


1 


2 


3 


4 


5 


8. 


Had activities that reinforced the content 


1 


2 


3 


4 


5 


9. 


Included appropriate examples 


1 


2 


3 


4 


5 


10. 


Was relevant to my needs 


1 


2 


3 


4 


5 


11. 


Had meaningful involvement of participants 


1 


2 


3 


4 


5 


12. 


Caused me to reflect on my practices 


1 


2 


3 


4 


5 


13. 


Caused me to examine some of my attitudes 


1 


2 


3 


4 


5 


14. 


Stimulated me to want to use the concepts, 
skills, and/or materials presented 


1 


2 


3 


4 


5 


15. 


Provided materials that will be useful to me in 
my continuing work as an external facilitator 


1 


2 


3 


4 


5 


16. 


Increased communication and collegiality 
with others 


1 


2 


3 


4 


5 


17. 


Increased my understanding of the role of 
the school district in school reform 


1 


2 


3 


4 


5 



Section B: Listed below are the objectives for the session(s) on the role of the district in reform. You 
are asked to rate each objective on two dimensions: 1) knowledge you gained during the session, and 2) 
your need for further information regarding the objective. 



Scale: 1 = None or not at all 


to 5 = Very much or a lot 




Knowledge gai.: 


Further need 


Enhanced my knowledge of the role of 
the district in school reform 


1 2 3 4 5 


12 3 4 


Increased my capacity to work with 
district staff 


1 2 3 4 5 


12 3 4 
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Academy ID #: 



AEL CSRD Academy for External Facilitators 
2000 Summer Institute 
Overall Institute Evaluation Form 



JVe are interested in continuously improving our services. Please help us to do so by taking a few 
moments to respond to the following items. Your replies will remain confidential and anonymous, so feel 
free to answer candidly. 



Section A. Please rate the following iteuis about 
the number that best corresponds to your reply. 



this Institute using the scales provided, circling 



1. Did AEL carry out planned activities at the i 
times scheduled? 


• ’ Never 

1 


2 


Some 

3 


4 


Always 

5 


2. How responsive were AEL staff and/or 
presenters to your requests for service and/or 
assistance during the Institute? 


Not at all 
1 


2 


Some 

3 


4 


Very 

5 


3. How knowledgeable were AEL staff and/or 
presenters? 


Not at all 
1 


2 


Some 

3 


4 


Very 

5 


4. How skilled were AEL staff and/or 
presenters in completing their tasks? 


Not at all 
1 


2 


Some 

3 


4 


Very 

5 


5. How well were your information needs met ; 
at this Institute? 


Not at all 
1 


2 


Some 

3 


4 


Very 

5 


6. How useful were the materials provided to 
you during this Institute? 


Not at all 
1 


2 


Some 

3 


4 


Very 

5 


7. How satisfied were you with the location of 
this event? 


Not at all 
1 


2 


Some 

3 


4 


Very 

5 


8. How satisfied were you with the format of 
this Institute? ' . 


Not at all 
1 


2 


Some 

3 


4 


Very 

5 


9. How satisfied were you with the meeting 
facilities at this event?; . | 


Not at all 

1 . , 


2 


Some 

3 


4 


Very 

5 


10. How useful do you anticipate this Institute 
will be to your work? _ 


Not at all 
1 


2 


Some 

3 


4 


Very 

5 



Section B. Using the scale provided, please rate the extent to which each of the follow ing Institute 
objectives were met. 



1. To what extent did the iistitute broaden yourj 
understanding of school reform and the change j 
process?, ‘ ■ ■ 

3. To what extent did you increase your : 
repertoire of strategies fpr helping schopls buildi 
their capacii^ for continuoiw improvement? J 



O 




Not at all 
1 2 



Some 
3 4 



A lot 
5 



Not at all Some 

\ \ ■. 2 3 



A lot 
4 5 
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4 To what extent did you assist to inform your 



state’s future reform efforts through sharing 
lessons learned? 


Not at all 
1 


2 


Some 

3 


4 


A lot 
5 


5. To what extent were you able to expand your^ 
resource network? 


: Not at all 

1 


2 


Some 

3 


4 


A lot 
5 


6. To what extent did you acquire an expanded , 
understanding of how change is planned, 
managed, and implemented? 


XT..4. -.4. -11 

iXUl iXi dii 
1 


2 


Sorric 

3 


4 


A 1-4. 

lUl 

5 


7. To what extent was your awareness of the 
connection between policy and practice in states 
broadened? , 


Not at all 

• v' . ^ 


2 


Some 

3 


4 


A lot 

■ .5 


8. To what extent did you develop new 
collegial relationships? 


Not at all 
1 


2 


Some 

3 


4 


A lot 
5 


9. To what extent did you learn from other 
participants? 


Not at all 
1 


2 


Some 

3 


4 


A lot 
5 


10. To what extent do you feel better prepared 
to work effectively with model developers and 
others providing services to school? 


Not at all 
1 


2 


Some 

3 


4 


A lot 
5 


1 1 . To what extent were your external 
facilitation skills augmented? 


Not at all 
1 


2 


Some 

3 


4 


A lot 
5 


1 2. To what extent was your knowledge of 
external facilitation enhanced? 


Not at all 
1 


2 


Some 

3 


4 


A lot 
5 


1 3 . To what extent was the opportunity to 
reflect on Year 1 with your state colleagues 
useful? 


Not at all 
1 


2 


Some 

3 


4 


A lot 
5 


14. To what extent was the opportunity to plan ; 
for Year 2 with your state colleagues useful? 


Not at all 
1 


2 


Some 

3 


4 


A lot 
5 



Section C. Please respond to the following prompts. 



1 . The high point of this Institute was . . . 



2. The Institute could have been improved by . . . 



3. Concerns I still have are . . . 



Thankyou! 



APPENDIX F: 

Completed Evaluation Standards Cheeklist 
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2000 Summer Institute Goals 



To help participants 

• broaden their understanding of school reform and the change process 

• increase their repertoire of strategies for helping schools build capacity for continuous 
improvement 

• inform their state’s future reform efforts through the distillation of lessons learned 

• expand their resource networks 



Inc. 

ERIC 



Academy for External Facilitators 
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Q Objectives 



Wednesday, August 2 




Clarify the purpose and scope of the 2000 Summer Institute 
Capture lessons learned from the first year of CSRD implementation 



D 

Q 

D 



Q*nn <5 m 

.1 i 1. 


— 


12:00 p.m 


2:00 p.m. 


- 


1 :00 p.m, 


1:00 p.m. 


— 


2:00 p.m. 



Airlie Dining Room 



2:00 
- 2:15 



0 

D 



p.m. - 

p.m. - 
Federal Room 



3:15 p.m. - 



Qi 

5:30 p.m. - 



2:15 p.m. 
3:15 p.m. 



5:30 p.m. 
6:30 p.m. 




;30 p.m. - 7:30 p.m. 

Airlie Dining Room 



Participant Arrival 
Check-in 

Registration 

Opening Luncheon 

Welcome 

Billie Hauser, Director of REL School Reform. AEL. Inc. 
Pamela Lutz, Executive Vice President. AEL. Inc. 

Break 

Plenary Session 

School Change: Who’s Saying What? Who’s Doing What? 
Betty Hale, Vice President, lEL 

Billie Hauser, Director of REL School Reform, AEL. Inc. 
David Wallace, Sequoyah Group 

CSRD Lessons Learned: Insights from State Partners 
Small Group Activity 



Break 

Check-In 

Dinner 



D 

D 

D 



I 

i 



1 
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Thursday, August 3 



Objectives 

1 . Expand understanding of how change is planned, managed, and implemented 

2. Foster broader awareness of the policy and practice connection in states 



7:00 a.m. - 8:00 a.m. 

Airlie Dining Room 

8:00 a.m. - 10:30 a.m. 

Federai Room 

10:30 a.m. - 10:45 a.m. 



Breakfast Buffet 
Plenary Session 

School Reform: Exploring Change Issues 
Break 



10:45 a.m. - 12:00 p.m. School Reform: Understanding Dimensions of Change 

Presenter, Andy Hargreaves, Co-director and Professor, 
Center for Educational Change, Theory, and Policy Studies 
in Education, University of Toronto 

12:00 p.m. - 1:00 p.m. Lunch 

Airlie Dining Room 

1:00 p.m. - 1:30 p.m. Insights: The Morning Session 

Federal Room 



1:30 p.m. - 3:00 p.m. 



3:00 p.m. - 5:00 p.m. 



State Meetings 
Kentucky 
Tennessee 
Virginia 
West Virginia 



West Room 
Garden Room 
Studio 
Foxes Den 



CSRD in the States: Sharing Insights with the 
Chief State School Officers 



5:00 p.m. - 6:00 p.m. Break 

6:00 p.m. - 6:00 p.m. Wine and Cheese Reception with the Chief State School Officers 

East Room 



7:00 p.m. - 8:00 p.m. Dinner 

Airlie Dining Room 



AF^ ^ 
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Friday, August 4 



Objectives 

1. Increase capacity to work more effectively with model developers and others providing services to schools 

2. Expand knowledge and skills of external facilitation for comprehensive school reform 



7:00 a.m. - 8:00 a.m. 

Aiiiie Dining Room 


Breakfast Buffet 


8:15 a.m. - 8:30 a.m. 

Federal Room 


Overview of the Day, Updates, Announcements 


8:30 a.m. - 


9:30 a.m. 


Plenary Session 

Facilitating Change in Schools: What Does it Take? 

Facilitator, Ben Perez, Principal, Transformations, Inc. 


9:45 a.m. - 


1 1 :00 a.m. 


Facilitating School Reform: Concurrent Breakout Sessions 
(See Schedule, next page) 


11:00 a.m. - 


11:15 a.m. 


Break 


11:15a.m. - 


12:30 p.m. 


Facilitating School Reform: Concurrent Breakout Sessions 
(See Schedule, next page) 


12:30 p.m. - 

East Room 


2:15 p.m. 


Lunch 

Working Collaboratively: A Dialogue with Model Developers 
John Batchelor (Success for All) 

Bob Blum (Onward Toward Excellence II) 

Ron Heady (Modern Red Schoolhouse) 

Connie Jones (Core Knowledge) 


2:30 p.m. - 


3:45 p.m. 


Facilitating School Reform: Concurrent Breakout Sessions 
(See Schedule, next page) 


3:45 p.m. - 


4:00 p.m. 


Break 


4:00 p.m. - 


5:00 p.m. 


State Meetings 

Kentucky West Room 

Tennessee Garden Room 

Virginia Studio 

West Virginia Foxes Den 


5:00 p.m. - 


6:00 p.m. 


Break 


6:00 p.m. - 


8:00 p.m. 


Cook Out/Barbecue 



Smokehouse 



[EL, Inc. 
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Facilitating School Reform: Concurrent Sessions 




CO 

in 



LO 

in 




I 

0 ( 1 = 

Q Objectives 



Saturday, August 5 



n 1 . Provide opportunity for states to reflect on Year I and plan for Year 2 
2. Celebrate and acknowledge contributions of external facilitators 



J 



D 

D 

D 

D 

D 



7:00 a.m. - 8:00 a.m. 

Airlie Dining Room 

8:30 a.m. - 10:30 a.m. 

Federal Room 



10:30 a.m. - 

1 

11:15 a. m. - 

! East Room 



11:15a.m. 
1:00 p.m. 



Breakfast Buffet 



What do I do on Monday? 

Facilitator, Ben Perez, Principal, Transformations, Inc. 



State Meetings 
Kentucky 
Tennessee 
Virginia 
West Virginia 



West Room 
Garden Room 
Studio 
Foxes Den 



Check out 

Recognition Luncheon 

Creating Change-Adept Schools 

Presenter, Phil Schlechty, President and CEO of the Center 
for Leadership in School Reform 

Safe Travel Home 




I 

I 



D 

0 

D 



AEL, Inc. 
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APPENDIX E: 



2000 Institute Agenda 



Checklist for Applying the Standards 



To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



Descriptor 

U 1 Stakeholder Identification 


The Standard was 
addressed 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


U2 Evaluator Credibility 










U3 Information Scope and Selection 










U4 Values Identification 










U5 Report Clarity 










U6 Report Timeliness and Dissemination 










U7 Evaluation Impact 










FI Practical Procedures 










F2 Political Viability 










F3 Cost Effectiveness 










PI Service Orientation 










P2 Formal Agreements 










P3 Rights of Human Subjects 










P4 Human Interactions 










P5 Complete and Fair Assessment 


y' 








P6 Disclosure of Findings 










P7 Conflict of Interest 


t/ 








P8 Fiscal Responsibility 










A1 Program Documentation 










A2 Context Analysis 










A3 Described Purposes and Procedures 










A4 Defensible Information Sources 


y 








A5 Valid Information 


y 








A6 Reliable Information 


y^ 








A7 Systematic Information 










A8 Analysis of Quantitative Information 










A9 Analysis of Qualitative Information 










A 10 Justified Conclusions 


y 








All Impartial Reporting 


y 








A 12 Metaevaluation 


y 









The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 



.request for evaluation plan/design/proposal 
. evaluation plan/design/proposal 
.evaluation contract 



-jki^evaluation report 
other: 



Name 









Date 









q|ai|oo 






Position or Title 



c^, 






Agency 






Address 



Relation to Document 






(e.g., author of document, evaluation team leader, external auditor, internal auditor) 
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U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reproduction Basis 




This document is covered by a signed "Reproduction Release 
(Blanket)" form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a "Specific Document" Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either "Specific Document" or "Blanket"). 
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